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Abstract 

The  US  DoD  has  tended  to  design  Command  &  Control  (C2)  systems  without 
consideration  for  them  to  interoperate  for  synergistic  effects  since  each  is  designed  for  one 
warfighting  function.  As  systems  have  grown  biologically  into  a  System  of  Systems, 
achievement  of  mission-level  effects  has  disappointed.  Architecting  the  C2  SoS  as  a  whole  is 
improbable.  However,  capabilities-based  acquisition  requires  interoperability  certification  based 
on  delivering  a  warfighter  capability  via  SoS.  Students  at  the  Naval  Postgraduate  School 
examined  this  problem.  Their  result  is  the  Joint  Capability  Command  and  Control  Management 
(JC3M)  system.  This  paper  summarizes  their  efforts.  A  systems  engineering  process  was 
applied  to  elicit  requirements,  create  and  simulate  alternative  solutions,  and  recommend  a 
solution  with  lifecycle  cost  estimates.  The  simulation  tools  selected  to  support  the  project  were 
CORE,  to  model  function  and  data  flow;  Arena,  for  timing  and  resource  utilization;  and  POW-ER 
(Project,  Organization,  Work  for  Edge  Research),  for  organizational  design  and  processes.  The 
use  of  these  tools  to  complement  each  other  is  unique.  Results  indicated  that  JTEM  Capability 
Test  Methodology  (CTM)  was  projected  to  perform  better  than  other  alternatives,  with  the 
median  LCC.  The  final  recommendation  is  to  monitor  JTEM  CTM  for  further  maturation  as  it 
promises  improvements  in  the  utility  of  C4I  SoS  evaluations. 

Keywords:  interoperability  assessment,  modeling,  systems  engineering 

Introduction 

Across  the  US  Department  of  Defense  (DoD),  early  C4I  systems  were  designed, 
acquired,  and  fielded  independently.  Each  addressed  a  single  warfighting  function,  such  as 
logistics,  fire  support,  or  intelligence.  Over  time,  warfighting  has  grown  in  complexity,  tempo, 
and  scope.  Complex  endeavors  are  characterized  by  participants  from  not  only  different 
services  but  also  from  different  functional  areas.  They  must  respond  with  agility  across  a 
spectrum  of  action  and  across  smeared  boundaries  between  tradition  levels  of  warfare.  The 
current  scenario  requires  a  network-centric  force,  which  in  turn  requires  true  C2  interoperability. 

Individual  C4I  systems,  most  not  designed,  acquired,  or  managed  as  a  collective 
enterprise,  are  being  integrated  as  such  and  are  forming  an  interdependent  entity — a  System  of 
Systems  (SoS) — in  which  emergent  behavior  dominates  and  capability  delivery  cuts  across 
system  boundaries.  System-level  acquisition  and  testing  only  result  in  individual  systems 
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meeting  specific  performance  requirements.  The  Joint  Interoperability  Test  Command  (JITC) 
tests  for  end-to-end  connections  “in  the  most  operationally  realistic  environment  possible” 

(rather  than  delivery  of  desired  capability)  to  assess  interoperability.  Successful  information 
exchange  results  in  “certification.”  This  is  the  baseline  system  for  DoD  interoperability 
certification.  However,  complex  interactions  of  effects  drive  changing  configurations  of  C4I  SoS 
with  no  formally  established  requirements  for  performance  evaluation.  Capability-based  testing 
of  a  SoS  is  not  well  understood.  However,  the  principle  to  ensure  interoperability  through 
testing  during  development  (National  Research  Council)  is  still  valid. 

The  baseline  interoperability  certification  process  is  inadequate  because  it  does  not 
address  how  the  actual  SoS  supports  complex  endeavors.  Recent  revision  to  the  Joint 
Capabilities  Integration  and  Development  System  emphasizes  that  true  interoperability  is 
characterized  by  “end-to-end  operational  effectiveness  [...]  for  mission  accomplishment”  (CJCS, 
2007).  Guidance  for  writing  Capability  Development  Documents  (CDD)  requires  Net-Ready  Key 
Performance  Parameters  that  assess  “the  net-ready  attributes  required  for  both  the  technical 
exchange  of  information  and  the  end-to-end  operational  effectiveness  of  that  exchange”  (DoD, 
2004).  This  is  consistent  with  the  NATO  definition  of  interoperability  (NATO,  2002)  and  that 
proposed  by  the  Software  Engineering  Institute  (Kasunic  &  Anderson).  Capability  Portfolio 
Managers  (DEPSECDEF,  2006,  September)  and  Functional  Capabilities  Boards  (CJCS,  2007) 
play  a  role  in  capabilities-based,  cross-program  interoperability.  Even  so,  no  system  can 
assess  the  capability  of  a  SoS  requiring  integration  of  functions  and  interfaces  across  multiple 
systems.  Thus,  a  JC3M  system  is  important  because  it  provides  a  process  for  test  planning  to 
verify  true  interoperability.  It  documents  traceability  between  capabilities  and  construction,  and 
it  provides  confidence  that  the  C4I  SoS  works. 

In  response  to  this  need,  the  Joint  Test  Evaluation  Methodology  (JTEM)  team  is 
addressing  Joint  SoS  interoperability  testing  at  the  Office  of  Secretary  of  Defense  (OSD)  level. 
Marine  Corps  Systems  Command  (MARCORSYSCOM),  the  acquisition  organization  for  the 
Marine  Corps,  is  approaching  the  issue  from  a  service  perspective.  MARCORSYSCOM  has 
tasked  the  Marine  Corps  Tactical  Systems  Support  Activity  (MCTSSA)  to  develop  Marine  Air 
Ground  Task  Force  (MAGTF)  C4I  Capability  Certification  Testing  (MC3T),  a  methodology  for 
managing  the  MAGTF  C4I  SoS  as  a  single  system.  MC3M  will  manage  the  MAGTF  C4I  SoS  as 
a  set  of  SoS-level  capabilities,  rather  than  as  a  fixed  hardware  or  software  baseline. 

NPS  students  assigned  to  the  JC3M  project  team  adopted  a  systems  engineering 
approach  to  the  problem  of  architecting  a  C4I  SoS  assessment  system  that  will  identify  desired 
effects-based  capabilities  and  ensure  that  the  system  being  tested  meets  those  requirements. 
The  JC3M  project  sought  a  lifecycle  balanced  solution  for  existing  test  organizations.  The 
processes  can  be  utilized  by  service  and  joint  test  agencies. 


Approach  Description 

The  student  design  team  adapted  several  systems  engineering  process  models  (Acosta 
et  al,  2007)  and  tailored  them  to  this  problem.  As  illustrated  in  Figure  1,  it  begins  with  identifying 
a  customer’s  needs  and  proceeds  through  several  phases  until  a  final  solution  is  recommended. 
One  can  see  this  is  a  modification  of  INCOSE’s  SIMILAR  (state  the  problem,  investigate 
alternatives,  model  the  system,  integrate,  launch  the  system,  assess  performance,  and  re¬ 
evaluate)  process  model  (INCOSE,  2007)  that  incorporates  elements  of  the  Systems 
Engineering  and  Design  Process  (Paulo,  2005)  taught  at  USMA  and  at  NPS. 
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Figure  1.  A  Tailored  Systems  Engineering  Process 

During  the  problem  refinement  phase,  research  into  the  problem  space  was  conducted, 
stakeholders  were  identified  and  interviewed,  functional  decomposition  was  started,  and  a  value 
system  was  developed.  Based  on  the  preliminary  functional  analysis  and  value  hierarchy, 
several  alternatives  were  created.  Those  alternatives  were  screened,  and  ultimately,  five 
alternatives  entered  the  modeling  and  simulation  phase.  The  predicted  performance  values 
generated  by  the  models  were  used  to  objectively  analyze  those  alternatives  by  comparing 
them  to  each  other  along  with  lifecycle  cost  estimates.  The  use  of  a  LCCE  as  part  of  the 
analysis  of  alternatives  in  this  problem  domain  is  vital.  Those  testers  and  test  planners  must  be 
paid  for;  it  matters  little  if  the  final  system  provides  the  best  solution  if  that  solution  is 
unaffordable.  Finally,  a  solution  was  recommended,  along  with  caveats.  Both  the  JTEM  project 
and  MC3T  project  will  make  use  of  those  recommendations. 

It  should  be  noted  that  this  team  did  an  excellent  job  connecting  values  identified  early 
by  stakeholders,  supported  by  a  thorough  functional  analysis.  They  integrated,  into  the  value 
hierarchy,  the  values  resulting  from  modeling  and  simulation  that  drove  the  final  decision 
process. 

Problem  Refinement  &  Functional  Analysis 

Developing  a  real  problem,  or  effective  need,  in  this  situation  proved  more  challenging 
than  anticipated.  Stating  the  central  issue  so  that  the  stakeholders  would  receive  some  utility 
from  the  final  solution  proved  slippery.  In  fact,  just  identifying  the  “right”  stakeholders  was  a 
challenge.  From  the  perspective  of  C4I  system  users,  any  process  to  certify  a  system  is 
interoperable  within  a  SoS  adds  value  when  that  certification  signifies  the  SoS’  ability  to  support 
the  complex  endeavor.  Verifying  that  it  conforms  to  technical  standards  and  that  it  can 
exchange  data  is  a  necessary,  but  not  sufficient,  prerequisite.  Whereas,  in  the  acquisition 
community,  a  program  manager  manages  resources  spent  for  certification.  If  test  results  are 
compared  to  criteria  outside  the  scope  of  his  or  her  program  or  are  not  explicitly  stated  in 
requirements  documents,  there  is  high  risk  with  little  gain.  The  test  community,  therefore,  finds 
itself  in  the  middle — being  the  honest  broker  representing  users  while  still  adding  value  to 
acquirers.  The  team  focused  on  the  test  community,  along  with  in-house  testers  inside  the 
acquisition  community,  as  primary  stakeholders.  The  final  list  included  the  Joint  Interoperability 
Test  Command  (JITC),  Marine  Corps  Operational  Test  and  Evaluation  Activity  (MCOTEA), 

Army  Test  and  Evaluation  Command,  Navy  Operational  Test  and  Evaluation  Force,  Air  Force 
Operational  Test  and  Evaluation  Center,  MCTSSA,  and  the  JTEM  Project  team  under  the 
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Director  of  Operational  Test  &  Evaluation.  As  this  team  was  mostly  composed  of  MCTSSA 
employees,  a  major  influence  was  the  new  MC3T  project,  which  provided  an  initial  primitive 
need  for  a  “system  that  defines  and  compares  System  of  Systems  performance  measures  to 
warfighter  needs  in  an  objective  and  measurable  way”  (Finn,  2007).  They  needed  a  system  that 
defined  threshold  values  for  C4I  SoS  performance  in  operational  warfighting  terms  and  then  a 
way  to  obtain  those  performance  measures. 

The  team  examined  the  larger  context  of  the  problem  to  find  the  underlying  need.  The 
team  researched  the  most  up-to-date  interoperability  certification  and  the  latest  direction  within 
the  DoD  that  examines  realizing  desired  capabilities.  While  the  existing  directives  and 
instructions  seem  clear  in  identifying  roles  and  responsibilities  in  a  traditional  sense,  little  light 
was  shed  on  the  root  of  the  issue.  All  stakeholders  were  queried  on  how  they  plan  a  C4I  SoS 
assessment,  what  resources  they  use  to  do  so,  how  component  systems  under  test  are 
identified,  how  performance  requirements  are  codified,  how  conflicts  are  resolved,  and  what 
metrics  they  use  to  assess  their  own  performance  (Acosta  et  al.,  2007).  The  written  questions 
sought  to  reveal  how  they  knew  they  succeeded  and  what  areas  were  most  ripe  for 
improvement.  The  responses  from  JTEM  and  JITC  were  professional,  insightful  and  frank. 

A  basic  functional  hierarchy  began  to  evolve  around  the  three  major  functions:  planning 
a  C4I  SoS  evaluation,  conducting  the  evaluation,  and  reporting  results.  The  identification  and 
definition  of  performance  threshold  values  was  of  primary  concern  and  all  stakeholders  seemed 
to  be  completely  satisfied  with  their  ability  to  execute  and  report  on  an  evaluation  event. 
Therefore,  the  problem  scope  was  focused  on  the  planning  phases.  Further  decomposition 
resulted  in  a  draft  functional  model,  shown  in  Figure  2  (Acosta  et  al.,  2007). 


Figure  2.  Initial  JC3M  Functional  Decomposition 

This  project  focused  entirely  on  function  1.0,  Plan  a  C4I  SoS  Evaluation.  Further 
functional  evaluation  identified  required  inputs  and  outputs  of  the  system,  process  activation  and 
termination,  and  evaluation  measures  for  each  of  the  lowest  level  functions. 

Eventually,  several  alternatives  were  to  be  compared  objectively.  The  basis  of  that 
comparison  was  how  well  they  achieved  the  functional  and  non-functional  requirements.  By 
combining  a  complete  functional  hierarchy  with  critical  non-functional  attributes  and  assigning 
evaluation  measures  to  each,  a  value  system  was  created.  This  classic  systems  engineering 
paradigm  completed  the  initial  requirements  analysis  work.  A  part  of  that  value  hierarchy — with 
only  the  critical  evaluation  measures  that  were  eventually  used  in  the  final  comparison  of 
alternatives — is  in  Figure  3.  This  is  a  small  sample  of  the  information  gained  through  the 
analysis.  However,  it  is  telling  because  it  codifies  how  designers  will  know  if  we  “got  it  right.” 
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Figure  3.  Part  of  JC3M  Value  Hierarchy 

(Acosta  et  al.,  2007) 

The  lighter-colored  boxes  indicate  the  evaluation  measures  that  defined  the  needs  for  a 
set  of  modeling  tools  and  that  would  drive  the  final  analysis.  A  more  complete  definition  for 
those  elements  is  provided  in  Table  1. 
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Table  1.  Evaluation  Measure  Details 


Percentage  of 
Traceable 
Measures 

Days  to 

Plan  Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

JC3M 

Function 

Define  Measures 
1.3.2 

Planning  Results 
1.4.3 

Planning  Results 
1.4.3 

Input  System 
Flexibility 

4.1 

Input  System 
Flexibility 

4.1 

Definition 

Alternative 
generated 
measures, 
traceable  to 
stakeholder 
requirements, 
divided  by  the 
number  of 

measures 
generated  by  the 
alternative. 

Elapsed  time  (in 
days)  of  planning 
for  C4I  SoS 
evaluation 

Assign  an  overall 
quality  level  to 
the  planning 
documents 
produced. 

Divide  percent 
change  in  labor 
hours  to  conduct 
planning  phase  of 
JC3M  by  the 
percent  change  in 
systems  under 
test. 

Divide  percent 
change  in  duration 
to  conduct 
planning  phase  of 
JC3M  by  the 
percent  change  in 
systems  under 
test. 

Ratio  level  data, 
from  0-100% 

Integer  count  >  0 
days 

Ordinal-Low, 
Medium,  High 

Ratio  level  data 
from  0-°° 

Ratio  level  data 
from  0-°° 

Rationale  and 
Relevance 

Identifies 
objectivity  of 
performance 
measures. 

Predicts  SoS 
evaluations  that 
can  be 

conducted  in  a 

Identifies 
predicted  utility  of 
alternative. 

Predicts  changes 
in  cost  of  SoS 
evaluation  based 
on  size. 

Predicts  changes 
in  duration  of  SoS 
evaluation  based 
on  size. 

Performance 

measures 

traceable  to 
doctrinal 

references  will  be 
perceived  as 
objective, 
increasing  the 
value  of  the 
evaluation. 

year. 

Alternatives  that 
permit  multiple 

SoS  evaluations 
generate  data  to 
support  fielding 
decisions  sooner. 

Quality  of  the 
planning 
products  drives 
the  overall  value 
of  the  alternative. 

Can  be  used  to 
determine  most- 
effective  alternative 
based  on  SoS  size. 

Can  be  used  to 
determine  most- 
effective 

alternative  based 
on  SoS  size. 

Design  Alternatives 

There  were  three  existing  alternatives  completed  or  in  the  final  stages  of  development  in 
response  to  the  problem  at  hand.  Additionally,  the  team  sought  to  architect  two  additional 
systems.  This  would  present  the  stakeholders  a  broad  range  of  possibilities,  while  keeping  the 
effort  required  for  modeling  and  simulation  manageable. 

The  first  of  the  known  alternatives  was  the  Federation  of  Systems  (FEDOS)  system  used 
at  MCTSSA  in  2005.  FEDOS  was  designed  to  assess  the  performance  of  C4I  systems  when 
assembled  into  the  MAGTF  C4I  SoS.  FEDOS  began  at  the  order  of  the  Deputy  Commander  for 
C4I  Integration  and  Interoperability  (C4II)  at  MARCORSYSCOM,  that  tasked  MCTSSA  to 
assess  SoS  and  systems  interoperability.  A  working  group  of  stakeholders  in  the  system 
developer  community  decided  which  systems  would  participate,  which  requirements  were  to  be 
tested,  and  the  schedule  of  events  to  include  test  planning,  test  conduct,  and  results  reporting. 

Because  the  MAGTF  C4I  SoS  was  not  designed  in  compliance  with  an  architecture, 
there  were  no  overarching  SoS  performance  measures  or  threshold  criteria.  This  lack  of 
doctrinal  performance  criteria  meant  that  MCTSSA  test  personnel  had  to  engage  in  long,  and  at 
times,  inconclusive  negotiations  with  stakeholders  to  define  threshold  values  that  were  used  to 
measure  performance  and  determine  if  components  passed  or  failed  the  test.  The 
MARCORSYSCOM  Product  Groups,  responsible  for  developing,  fielding,  and  supporting  C4I 
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systems,  were  not  ordered  to  participate  in  FEDOS,  and  a  passing  grade  was  not  required  for  a 
milestone  decision.  It  was  perceived  as  a  no-win  situation  for  Product  Groups:  after  a  system 
had  successfully  passed  Operational  Tests  by  demonstrating  compliance  with  system-level 
performance  requirements  in  their  respective  ODD  or  equivalent,  FEDOS  tested  component 
systems  in  ways  they  had  not  been  designed  for,  but  would  be  used  in  the  field.  The  acquisition 
community’s  perception  was  that  FEDOS  was  a  risk  with  no  off-setting  benefit.  Despite  this 
shortcoming,  FEDOS  was  relatively  successful  as  the  first  USMC  event  specifically  designed 
from  the  beginning  as  a  SoS  evaluation 

Because  FEDOS  is  the  only  alternative  solution  that  has  been  used  by  a  C4I  test 
organization  for  a  true  SoS  event,  it  was  considered  the  “status  quo”  or  baseline  JC3M 
alternative  solution.  As  with  all  good  analyses  of  alternatives,  the  first  option  to  consider  is  “do 
nothing,”  or,  in  this  case,  “do  it  like  FEDOS.” 

The  second  alternative  was  MAGTF  C4I  Capability  Certification  Test  (MC3T)  developed 
at  MCTSSA  as  a  replacement  for  FEDOS.  Other  participants  in  MC3T  development  include  the 
Space  and  Naval  Warfare  Center  (SPAWAR)  Systems  Center  in  Charleston,  S.C.,  and  the 
Marine  Corps  Combat  Development  Command  (MCCDC).  More  importantly,  representatives  of 
the  MARCORSYSCOM  Product  Groups  actively  participated.  Product  Group  representatives 
defined  a  "Capabilities  Package"  complete  with  system  requirements  and  DoD  Architecture 
Framework  documents  that  depict  the  systems  under  their  cognizance.  MCTSSA  analyzed  the 
Capabilities  Package  and  produced  a  Consolidated  Requirements  Assessment  (CRA).  The 
CRA  was  an  agreement  between  the  stakeholders  on  what  needed  to  be  tested,  the  required 
resources,  and  the  Information  Assurance  compliance  requirements.  Once  the  CRA  was 
approved,  MCTSSA  produced  a  Technical  Proposal.  The  Technical  Proposal  defined  the 
technical  solution  that  the  IPT  proposed  in  order  to  meet  the  requirements  in  the  Consolidated 
Requirements  Assessment  (CRA),  including  staffing,  C4I  systems  architecture  design, 
monitoring  network  architecture  design,  test  cases,  data  capture  and  analysis  plan,  information 
assurance  plan,  and  risk  assessment.  The  Technical  Proposal  is  confirmed,  becoming  the 
Technical  Solution,  which  makes  up  nearly  90%  of  the  Test  Plan,  includes  detailed  test 
procedures  with  reference  documentation.  The  most  promising  aspect  of  MC3T  is  that  MCCDC 
and  MARCORSYSCOM  have  developed  truly  integrated  architecture  framework  products.  The 
operational  activities  doctrinally  defined  in  the  Marine  Corps  Task  List  are  explicitly  supported  by 
specific  systems  working  together.  The  idea  that  form  should  follow  function  in  designing  for 
network-centric  effects-based  operations  is  consistent  with  the  latest  direction  for  architectures 
(DoD,  2007). 

The  third  alternative  was  JTEM’s  Capability  Test  Methodology  (CTM).  The  purpose  of 
JTEM  is  to  “develop,  test,  and  evaluate  M&P  (Methods  and  Processes)  for  defining  and  using  a 
distributed  LVC  (Live,  Virtual,  and  Constructive)  joint  test  environment  to  evaluate  system 
performance  and  joint  mission  effectiveness  [...]  focus  on  developing  and  enhancing  M&P  for 
designing  and  executing  tests  of  SoS”  (JTEM,  2007b).  Figure  4  is  an  IDEF0  representation  of 
the  CTM  process. 
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Capability  Dev,  Document  (CDD) 
Analytical  Baseline  (Scenario,  CONORS,  Data) 


Approved  AoA  Plan 


Initial  Capabilities  Document  (ICD) 


Figure  4.  JTEM  CTM  in  IDEFO 

(Acosta  et  al.,  2007) 

One  of  the  more  promising  aspects  of  JTEM’s  CTM  is  that  test  characterization  explicitly 
examines  requirements  from  families  of  CDDs  in  the  context  of  missions  based  on  the  Universal 
Joint  Task  List  (UJTL)  (CJCS,  2002)  and  Combatant  Command  standing  operations  plans  and 
orders.  More  detailed  descriptions  can  be  found  in  JTEM’s  Joint  Test  and  Evaluation  (JT&E), 
Capability  Test  Methodology  (CTM)  Method  and  Process  (M&P)  Model  Description  (JTEM, 

CTM,  M&P).  The  complexity  of  scenarios  developed  for  the  LVC  test  environment  reflects  real- 
world  complex  military  action  involving  disparate  forces  executing  closely  linked  complicated 
tasks,  including  operations  other  than  war. 

Two  new  alternatives  that  offer  significant  differences  from  the  existing  systems  were 
developed.  The  classic  morphological  box  (Zwicky  process)  was  applied  and  guided  by  the 
high-level  functions  identified  earlier  and  then  used,  in  part,  to  identify  evaluation  measures. 

Nine  alternatives  were  initially  defined.  Through  several  screening  iterations  and  re-evaluation 
against  the  root  problem,  only  two  remained:  “Systems  Capabilities  Review”  (SCR  Alternative) 
and  “Functional  Capabilities  Board”  (FCB  Alternative). 

The  Systems  Capabilities  Review  (SCR)  alternative  combines  two  of  the  original  nine 
alternatives.  It  is  composed  of  a  group  of  stakeholders:  C4I  SoS  user  representatives,  test 
agency  representatives,  system  developers  and  program  managers.  The  test  agency 
representative  chairs  the  group,  which  meets,  as  required,  to  support  a  C4I  SoS  evaluation,  at 
the  Systems  Command  level.  Inputs  to  SCR  include  source  documents  such  as  Capabilities 
Development  Documents  (CDD),  Operational  Requirements  Documents,  Test  and  Evaluation 
Master  Plans  (TEMP),  Concept  of  Operations  documents,  Joint  Integrating  Concepts,  Joint 
Operating  Concepts,  and  system  level  metrics.  First,  the  SCR  reviews  SoS  capabilities 
specifications,  examines  the  systems  engineering  artifacts  already  created  (such  as  supporting 
DoD  Architecture  Framework  documents  and  technical  performance  measures)  and  creates  a 
list  of  implied  and  stated  SoS  capabilities.  Next,  the  SCR  reviews  system-level  documents  and 
creates  a  system-level  capabilities  list.  Third,  the  SCR  maps  system-level  capabilities  to  SoS 
evaluation  measures.  The  SCR  identifies  gaps  in  the  evaluation  measure  list  and  creates  the 
balance  of  evaluation  measures  necessary  to  evaluate  the  performance  of  the  C4I  SoS.  Figure 
5  illustrates  how  SCR  performs  the  JC3M  subfunction  1 .3.2  “Define  Measures.” 
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SCR  Alternative 


Figure  5.  SCR  Alternative  Sub-functions 

(Acosta  et  al.,  2007) 

The  Functional  Capabilities  Board  (FCB)  alternative  relies  on  an  existing  group — the 
JCIDS  C2  Functional  Capabilities  Board — to  define  the  performance  measures  of  the  SoS.  The 
existing  role  of  FCB  is  to  perform  “organization,  analysis,  and  prioritization  of  joint  warfighting 
capabilities  within  an  assigned  functional  area”  (CJCS,  2007).  Inputs  to  the  FCB  Alternative 
include  the  UJTL  and  subsets,  Concept  of  Operations  (CONOPS)  documentation,  acquisition 
program  documentation,  and  system  trouble  reports.  The  additional  effort  proposed  in  this 
alternative  represents  an  increase  in  the  work  performed  by  the  C2  FCB  but  is  in  the  same 
functional  area  and  engages  in  the  similar  tasks.  Unlike  the  SCR,  the  FCB  meets  on  demand, 
rather  than  as  required,  to  support  SoS  evaluations.  First,  the  FCB  will  identify  the  configuration 
of  the  SoS  by  determining  the  component  systems.  Next,  the  FCB  will  identify  the  SoS 
capabilities.  SoS  CONOPS  are  reviewed  to  determine  evaluation  measures.  Finally,  the  FCB 
will  generate  the  SoS  evaluation  measure  list  for  use  in  C4I  SoS  evaluations.  As  the  systems 
under  the  cognizance  of  the  Joint  Command  &  Control  Capability  Portfolio  Manager  are 
explicitly  listed  (DEPSECDEF,  2006,  September),  their  participation  in  this  alternative  would  be 
required.  The  FCB,  under  JCIDS,  has  a  long-term  mandate,  and  provides  a  short-term  solution 
to  the  lack  of  SoS  performance  measures.  The  relationship  between  the  FCB  and  C4I  test 
organizations  and  the  list  of  subtasks  needed  to  complete  the  Define  Measures  task,  is 
illustrated  in  Figure  6.  Because  the  FCB  is  external  to  the  test  organization,  some  analysis  of 
the  performance  measures  generated  by  the  FCB  will  be  necessary.  Additionally,  it  is 
understood  that  a  working  group  within  the  FCB  would  perform  the  required  analysis. 
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FCB  Alternative 


FCB  Team 
external  to 
the  test 
agency 


1.1  Assesses  issues  to 
support  JROC 
recommendations 


1.2  Assess  DOTMLPF 
Change 

Recommendation  (DCR)s 
using  the  following  criteria 


1.3  Make 

recommendations  to  their 
respective  FCBs 
concerning  proposal 
content  and  suitability  for 
presentation  to  the  JCB 
and  JROC. 


Figure  6.  FCB  Alternative  Sub-functions 

(Acosta  et  al.,  2007) 

Both  of  these  new  alternatives  developed  by  the  JC3M  team  rely  on  supporting 
integrated  architectures  and  CONOPS  documentation,  in  addition  to  documentation  normally 
examined  as  part  of  C4I  interoperability  test  preparation.  The  difference  between  these 
alternatives  is  in  the  approach  taken  to  complete  process  1 .3.2  “Define  Measures”  in  the  JC3M 
Functional  Hierarchy.  The  SCR  alternative  incorporates  all  tasks  as  part  of  the  test  planning 
process.  The  FCB  Alternative  utilizes  an  external  team  that  meets  year-round  to  provide 
capability  measures  to  the  test  agency. 

Five  alternatives  had  now  been  defined  in  some  detail,  as  well  as  evaluation  measures 
to  be  used  to  compare  those  alternatives.  Only  determining  the  actual  values  or  values 
obtained  from  simulation  models  for  each  alternative  remained. 

Modeling  &  Results 

Modeling  and  simulation  were  used  extensively  in  this  project.  With  the  exception  of 
FEDOS,  no  other  alternative  under  consideration  existed.  The  only  means  to  gather 
performance  data  in  support  of  decision-making,  short  of  “building”  each  alternative,  was 
through  simulation.  It  was  the  most  cost-effective  means  to  obtain  the  required  evaluation 
measures  in  a  repeatable  and  objective  fashion.  Several  modeling  tools  were  used  to  generate 
the  necessary  data.  Figure  7  illustrates  which  tools  were  used  to  obtain  the  evaluation 
measures,  which  in  turn  supported  later  cost-benefit  analysis. 
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Figure  7.  JC3M  M&S  Supporting  Evaluation  Measures 

(Acosta  et  al.,  2007) 


Models  of  each  alternative  were  built  based  on  the  functional  architectures  already 
created.  Elements  unique  to  their  physical  instantiations  were  added.  In  other  words,  complete 
functional  models  in  IDEFO  were  created  with  Vitech’s  CORE  to  support  the  simulation  models 
built  in  Arena  and  POW-ER  (Project,  Organization,  and  Work  for  Edge  Research).  Within  Arena 
and  POW-ER,  the  attributes  that  differentiated  the  alternatives  from  each  other —  organizational 
structure,  relationships  with  external  systems,  and  processing  of  certain  inputs — were  included. 
The  IDEFO  view  of  the  systems  actually  proved  insightful  in  terms  of  explicitly  describing  the 
relationship  between  the  functions,  at  all  levels  of  abstractions,  in  terms  of  their  inputs  and 
outputs.  The  models  were  executed  by  providing  input  to  simulate  a  system  under  test  along 
with  its  supporting  information.  The  results  of  several  iterations  with  variations  in  the  input  data 
sets  were  gathered  and  used  to  populate  the  table  of  evaluation  measures  with  raw  data.  The 
“off-line  evaluation”  indicated  the  use  of  desk-top  evaluation  by  test  and  development 
community  representatives,  similar  to  the  JTEM  Rock  Drills.  It  could  be  considered  a  kind  of 
human-in-the-loop  simulation  or  just  another  kind  of  model  or  prototype  that  has  been  used 
successfully  in  this  problem  domain  (JTEM,  2007b). 

POW-ER  is  a  project  organization  modeling  and  simulation  tool  that  integrates 
organizational  and  process  views.  POW-ER  was  developed  via  the  Virtual  Design  Team  (VDT) 
computational  modeling  research  at  Stanford  University.  POW-ER  addresses  organizational 
elements  that  impact  the  ability  to  work  effectively,  including  policies  and  structures  (culture, 
communication,  decisions,  meetings);  staffing,  hiring,  and  training  needs  for  workforce  plans. 
Using  POW-ER,  the  team  modeled  the  organizational  structure,  the  relationship  between 
individuals  within  those  organizations,  and  individual  task  allocations.  Use  of  CORE  to  support 
functional  analysis  proved  most  helpful  as  it  allowed  the  modelers  to  represent  the  same 
functional  architecture  in  the  refined  IDEFO  models  as  a  functional  flow  in  FFBD  format.  That 
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allowed  the  creation  of  PERT-like  sequencing  of  tasks  required  when  modeling  work  processes 
in  POW-ER.  POW-ER’s  ability  to  predict  and  analyze  backlogs  proved  useful  designing  and 
troubleshooting  alternative  models  because  it  allowed  the  team  to  identify  backlogs  in  the 
workflow  of  models.  The  analysis  of  backlogs  in  the  workflow  enabled  the  team  to  identify  the 
optimized  arrangement  of  tasks  and  personnel  for  FCB  and  SCR  since  they  were  created  for 
this  project.  No  such  changes  were  made  to  the  other  alternatives.  Based  on  modeler-defined 
parameters,  such  as  the  amount  of  effort  required  for  each  task,  the  number  of  full-time 
equivalents  available  with  appropriate  skills  and  number  of  hours  in  a  work-week,  the  POW-ER 
simulation  tool  can  calculate  a  project’s  duration  based  on  simulated  duration.  Simulated 
duration  factors  the  “hidden  work”  that  traditional  Critical  Path  Method  does  not.  The  “hidden 
work”  associates  an  amount  of  rework  that  delays  into  each  task  based  upon  random  variables 
described  for  each  task  by  the  modeler.  The  simulated  duration  provided  the  number  of  days  to 
plan  an  evaluation  for  each  alternative  (Acosta  et  al.,  2007). 

Arena  is  a  commercial  tool  available  from  Rockwell  Automation.  It  provides  a  numerical 
evaluation  of  a  system  by  imitating  the  system’s  operations  or  characteristics  over  time.  Arena 
allowed  the  team  to  conduct  numerical  experiments  in  order  to  predict  the  behavior  of  an 
alternative,  given  a  set  of  conditions.  Two  evaluation  measures  required  assessing  the  changes 
in  output  as  a  function  of  the  changes  in  inputs:  Elasticity  of  Labor  and  Elasticity  of  Duration. 
Arena  allowed  the  team  to  run  simulations  on  the  alternative  models  with  varying  sets  of  inputs. 
Those  input  data  sets  represent  the  number  of  systems  with  their  associated  documentation 
that  a  SoS  test  event  would  typically  cover.  The  baseline  data  set  was  the  group  of  systems 
used  during  the  FEDOS  event.  It  included  over  90  systems,  which  included  AFATDS,  EPLRS, 
GCCS-J,  SINCGARS  and  TBMCS.  There  were  14  SoS  capabilities  examined,  including  blue 
force  common  operational  picture,  call  for  fire,  common  logistics  and  theater  ballistic  missile 
tracking.  Variation  in  the  input  data  set  was  accomplished  by  changing  the  number  of  individual 
systems,  the  number  of  old  SoS  capabilities,  and  the  number  of  new  SoS  capabilities  under  test 
for  each  data  set.  The  same  input  data  set  was  used  for  one  run  of  each  alternative,  enabling  a 
true  head-to-head  comparison.  The  model  in  Arena  was  designed  so  that  the  subprocess  tasks 
would  vary  in  duration,  based  on  varying  the  input  systems  under  test.  Thus,  Arena  displayed 
the  output  changes  of  the  entire  alternative  process  that  corresponded  to  each  of  the  varying 
inputs.  The  output  changes  (as  a  percent  of  the  baseline),  compared  to  the  percent  change  of 
the  input  became  the  values  for  elasticity  of  duration  and  elasticity  of  labor  (Acosta  et  al.,  2007). 

The  models  were  validated  against  actual  data  from  the  FEDOS  event  of  2005.  Since 
the  original  labor  hour  timesheets  for  planning  that  event  were  available,  validating  the  models 
was  relatively  simple.  The  FEDOS  process  model  was  built  in  CORE,  which  supported  the 
more  elaborate  models  in  POW-ER  and  Arena.  Then,  the  outputs  were  compared  to  the 
appropriate  actual  data  from  FEDOS.  The  number  of  labor  hours  and  calendar  day  predictions 
from  Arena  and  POW-ER  were  within  1%  of  the  actual  values  (Acosta  et  al.,  2007). 

Figure  8  summarizes  the  entire  simulation  process,  including  inputs  and  output  values. 
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Figure  8.  JC3M  Modeling  Overview 

(Acosta  et  al.,  2007) 

This  study  represents  the  first  time  these  modeling  tools  were  used  together  to 
complement  each  other.  The  simulations  predicted  key  parameters  of  each  alternative  design. 
Without  such  an  approach,  no  objective  or  repeatable  means  to  compare  the  alternatives 
against  the  requirements  in  those  areas  would  have  been  possible.  There  is  a  high  degree  of 
confidence  in  the  computer-based  measures  because  the  results  for  the  FEDOS  models  were 
validated  against  known  historical  data  and  the  other  models  used  elements  from  the  data, 
based  on  a  task  mapping  from  each  alternative  back  to  the  FEDOS  process. 

There  were  still  two  evaluation  measures  that  could  not  be  determined  by  computer- 
based  simulation:  percent  traceable  measures  and  quality  of  planning  outputs.  The  team  was 
able  to  engage  SMEs  from  several  NAVSEA  and  NAVAIR  field  activities  to  participate  in 
assigning  a  value  for  quality  of  planning  products.  The  team  assembled  and  then  presented 
with  all  five  alternatives.  They  were  then  allowed  to  ask  questions  in  order  to  ensure  clear 
understanding  of  how  each  process  worked  along  with  built-in  limitations.  Each  SME 
responded  to  specific  questions  about  the  predicted  quality  of  planning  products  coming  from 
each  process,  with  regard  to  their  effectiveness  in  examining  interoperability  within  a  SoS, 
conformance  to  standards,  and  usability.  The  responses  were  based  on  a  4-point  Likert  scale 
for  each  alternative.  Percent  of  traceable  measures  was  more  simple  to  determine  once  a  key 
assumption  was  accepted.  A  proxy  was  defined  as  the  number  of  authoritative  sources 
considered,  divided  by  the  total  number  of  authoritative  sources  available.  This  assumption  is 
valid  if  there  is  a  linear  relationship  (as  a  set)  between  the  number  of  measures  created  and  the 
number  of  sources  used  in  creating  those  measures. 

The  final  listing  of  the  raw  scores  is  provided  in  Table  2. 
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Table  2.  Raw  Evaluation  Measures 


Percentage  of 
Traceable 
Measures 
(%> 

Days  to  Plan 
Evaluation 

(Days) 

Quality  of 
Planning 
Outputs 

(1-4  Likert 
Scale) 

Elasticity  of 
Labor 

(unit  less) 

Elasticity  of 
Duration 

(unit  less) 

Ideal  Value 

100% 

Less  is  better 

4  is  Ideal 

Less  is  better 

Less  is  better 

FEDOS 

0 

140 

3.17 

0.87 

0.87 

MC3T 

72 

121 

3.25 

0.78 

0.78 

JTEM  CTM 

92 

73 

3.42 

1.04 

0.83 

SCR 

92 

158 

3.00 

0.98 

0.98 

FCB 

88 

127 

2.75 

0.72 

0.72 

The  extremely  short  duration  to  plan  an  event  for  the  JTEM  CTM  process  should  be 
noted.  This  is  to  be  expected  because  of  that  system’s  reliance  on  SMEs  in  so  many  different 
fields,  which  minimizes  cross-checking  with  multiple  stakeholders.  On  the  other  hand,  the 
JTEM  CTM  elasticity  of  labor  was  the  worst. 

Considering  so  many  measures,  how  could  a  single  “best”  alternative  be  found?  The 
team  chose  to  apply  classic  multi-attribute  utility  theory  (MAUT).  While  MAUT  has  its  well- 
documented  limitations,  it  presents  a  means  to  compare  the  alternatives  on  a  single  weighted 
sum  of  utilities  associated  with  each  evaluation  measure.  Raw  scores  are  converted  to  a  value 
or  utility  score;  that  value  is  then  multiplied  by  its  global  weight,  and  the  resulting  weighted 
values  are  summed  to  an  overall  value.  The  same  SMEs  who  participated  in  the  process  to 
obtain  planning,  product-quality  figures  also  participated  in  the  process  to  determined  value 
functions  and  swing  weights.  It  should  be  noted  that  this  team  used  the  mathematically  rigorous 
Wymorian  standard  scoring  functions  for  value  curves  to  convert  raw  scores  to  utility. 
Additionally,  they  were  very  precise  about  their  application  of  swing  weights  and  rigor  of  the 
analytical  hierarchy  process  to  obtain  weights  (Acosta  et  al.,  2007).  So,  the  weaknesses 
inherent  in  MAUT  were  minimized  via  these  tools  and  techniques.  The  final  total  scores  are 
shown  in  Table  3. 


Table  3.  Overall  Utility  of  the  Alternatives 


Percentage 
of  Traceable 
Measures 

Days  to  Plan 
Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity  of 
Labor 

Elasticity  of 
Duration 

Overall 

Utility 

(0-1) 

FEDOS 

0.00 

0.04 

0.39 

0.06 

0.14 

0.63 

MC3T 

0.02 

0.05 

0.39 

0.07 

0.16 

0.71 

JTEM  CTM 

0.24 

0.06 

0.40 

0.04 

0.15 

0.89 

SCR 

0.24 

0.02 

0.37 

0.05 

0.10 

0.79 

FCB 

0.22 

0.05 

0.34 

0.08 

0.18 

0.87 

The  last  step  in  the  process  to  consolidate  the  elements  of  the  alternatives  was  to  create 
a  lifecycle  cost  estimate  (LCCE)  for  each  alternative.  All  costs  associated  with  development, 
implementation,  operations  and  support  through  disposal  and  transition  were  estimated.  Actual 
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data  from  the  FEDOS  event,  to-date  actual  costs  and  to-completion  estimates  (directly  from 
their  respective  project  managers)  for  development  of  JTEM  CTM  and  for  development  of  MC3T 
were  relatively  easy  to  capture,  once  complete  definitions  for  those  phases  and  cost-breakdown 
structures  were  developed.  Because  the  SCR  and  FCB  alternatives  were  similar  to  MC3T  in 
scope  and  effort,  development  costs  were  based  on  the  MC3T  numbers.  As  operations  and 
support  for  such  a  system  is  dominated  by  labor  costs,  the  annual  cost  for  each  alternative  was 
based  on  applying  the  prevailing  man-hour  rates  to  the  labor  hour  counts  from  the  POW-ER 
models.  Disposal  and  transition  costs  were  assumed  to  be  the  same  for  each  alternative 
because  those  efforts  were  practically  identical  in  terms  of  level  of  effort  and  duration.  Table  4 
summarizes  the  LCCE  for  each  alternative. 

Table  4.  LCCE  Summary 


Lifecycle  Year 

Total  Cost 

Alternatives 

1 

2 

3 

4.. .9 

10 

<$) 

FEDOS 

Development 

0 

0 

0 

0 

0 

0 

Implementation 

1,052,527 

0 

0 

0 

0 

1,052,527 

Operational  &  Maint. 

0 

419,497 

419,497 

419,497 

2,200 

3,908,178 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

1,052,527 

419,497 

419,497 

419,497 

52,200 

5,010,706 

MC3T 

Development 

0 

0 

0 

0 

0 

0 

Implementation 

1,169,414 

0 

0 

0 

0 

1,169,414 

Operational  &  Maint. 

0 

525,537 

525,537 

525,537 

2,200 

4,756,500 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

1,169,414 

525,537 

525,537 

525,537 

52,200 

5,975,913 

JTEM  CTM 

Development 

1,030,000 

2,470,000 

0 

0 

0 

3,500,000 

Implementation 

0 

0 

1,169,414 

0 

0 

1,169,414 

Operational  &  Maint. 

0 

0 

0 

558,535 

2,200 

2,253,410 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

1,030,000 

2,470,000 

1,169,414 

558,535 

52,200 

6,972,824 

FCB 

Development 

1,021,835 

0 

0 

0 

0 

1,021,835 

Implementation 

1,301,282 

0 

0 

0 

0 

1,301,282 

Operational  &  Maint. 

0 

650,223 

650,223 

650,223 

2,200 

5,753,985 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

2,323,117 

650,223 

650,223 

650,223 

52,200 

8,127,101 

SCR 

Development 

952,007 

0 

0 

0 

0 

952,007 

Implementation 

1,169,414 

0 

0 

0 

0 

1,169,414 

Operational  &  Maint. 

0 

624,451 

624,451 

624,451 

2,200 

5,547,811 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

2,121,421 

624,451 

624,451 

624,451 

52,200 

7,719,232 
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The  JC3M  team  determined  the  most  expensive  alternative  was  the  FCB  Alternative,  at 
a  cost  of  $8.13  million  over  the  10-year  projected  lifecycle.  The  team  calculated  the  cost  of  FCB 
as  a  cost  to  the  DoD.  While  the  senior  SMEs  who  generate  the  performance  measures  do  not 
charge  their  efforts  directly  to  a  C4I  test  organizations,  their  time  and  effort  is  a  cost  to  the  DoD. 
The  team  determined  that  MC3T  was  estimated  to  cost  approximately  $960,000  more  than 
FEDOS,  which  it  replaced.  While  this  is  nearly  a  20%  difference,  the  increase  can  be  directly 
attributed  to  the  increase  in  scope,  duration,  and  level  of  effort  involved  in  MC3T,  which 
anecdotally  supported  the  increased  cost  of  MC3T  (Acosta  et  al.,  2007).  More  importantly,  the 
development  cost  for  JTEM-CTM  is  the  largest  (its  development  is  spread  over  several  years). 
However,  the  O&S  costs  are  the  lowest.  This  result  is  significant  because  a  test  agency  (or  test 
branch  within  a  development  agency)  deciding  between  these  options  would  incur  only  the 
costs  to  implement  such  an  option  and  then  would  reap  the  benefit  of  keeping  annual  costs  very 
low. 


Recommendations 

A  complete  analysis  of  the  alternatives  based  on  the  preceding  data  was  conducted  to 
determine  the  “best”  alternative.  That  is,  which  alternative  is  projected  to  provide  the  greatest 
utility  for  the  cost?  Figure  9  summarizes  the  results.  Again,  the  utility  is  a  weighted  sum  of 
several  different  attributes,  all  tied  directly  to  the  overall  goal  of  ensuring  testing  for  true 
interoperability,  which  is  a  pre-requisite  for  any  C2  SoS  supporting  a  disparate  networked  force. 


Life  Cycle  Cost  Estimate  ($MIL) 


Figure  9.  Utility  versus  LCC 

(Acosta  et  al.,  2007) 

The  JTEM  CTM  process  is  projected  to  perform  slightly  better  than  the  other  options  and 
maintains  a  LCCE  less  than  the  two  other  alternatives  with  the  closest  utility  scores.  The 
attributes  that  drive  this  performance  are  the  number  of  days  to  plan  an  evaluation,  the  quality  of 
planning  products  and  the  percentage  of  traceable  measures.  It  should  also  be  noted  that  a 
nearly  straight  line  could  be  drawn  between  FEDOS,  MC3T  and  FCB.  That  leaves  the  SCR 
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Alternative  below  the  line  and  JTEM  CTM  above  it.  However,  the  better  way  to  examine  this 
figure  is  to  consider  an  efficient  frontier  of  utility  for  every  cost  value.  A  linear  frontier  is  formed 
by  a  line  connecting  the  points  for  FEDOS,  MC3T,  and  CTM.  Thus,  the  FCB  and  SCR  points 
are  “below”  that  line — meaning  they  are  less  efficient  and  dominated  by  CTM. 

It  must  be  noted  that  there  is  some  difference  in  the  confidence  we  have  in  the 
performance  measures.  Because  FEDOS  and  MC3T  were  used  in  actual  full-scale  SoS  test 
events,  their  performance  is  based  on  historical  documentation.  JTEM  CTM’s  performance 
measures  are  based  on  desk-top  simulations  called  “rock  drills,”  in  which  test  community 
personnel  exercised  certain  aspects  of  the  system  in  an  artificial  scenario.  Additionally, 
members  of  the  JTEM  team  participated  in  this  study,  which  validates  nearly  every  aspect  of 
JTEM  CTM  that  was  considered  and  confirms  the  expected  simulation  output.  The  results  from 
the  SCR  and  FCB  alternatives  were  purely  from  the  simulation.  However,  the  simulation  was 
based  on  modifying  parts  of  models  validated  through  FEDOS  data. 

With  regard  to  cost,  similar  logic  can  be  applied.  Those  numbers  from  FEDOS  and 
MC3T  are  based  on  actual  costs.  The  cost  estimates  for  the  other  alternatives,  dominated  by 
the  labor  of  annual  operations,  were  driven  by  the  simulation  output  for  number  of  labor  hours. 

In  spite  of  the  differences  in  confidence  levels,  the  overall  results  should  be  considered 
valid.  The  JTEM  CTM  had  the  median  LCCE,  with  the  lowest  O&S  cost.  This  is  significant 
because  O&S  is  a  recurring  cost,  borne  by  every  C4I  test  organization  that  implements  one  of 
the  alternatives.  Development  costs  of  JTEM  CTM  are  the  largest  portion  of  its  LCCE — a 
nonrecurring  cost  borne  by  OSD  and  not  borne  by  any  single  C4I  test  organization. 


Summary  &  Next  Steps 

This  team  was  the  first  to  apply  a  disciplined  systems  engineering  process  to  the 
problem  of  re-engineering  the  business  of  testing  for  C4I  interoperability  certification.  The  JTEM 
project  is  the  only  other  organization  to  examine  this  issue  from  the  perspective  of  optimizing  a 
lifecycle-balanced  solution  to  meet  explicitly  stated  and  quantifiable  needs.  No  group  has 
applied  an  integrated  set  of  computer-based  simulation  tools  to  quantitatively  predict  the 
performance  of  competing  options  and  compare  that  performance  to  lifecycle  cost.  Knowing 
that  C4I  systems  never  perform  in  a  vacuum,  but  always  interoperate  as  part  of  a  larger  SoS, 
developers  and  testers  will  benefit  from  the  results  of  this  study.  Ensuring  interoperability 
across  services  and  between  civil  authorities  and  multinational  organizations  begins  with  an 
effects-based  approach.  Only  by  testing  for  interoperability  against  performance  measures  that 
are  linked  to  desired  effects  in  the  battle-space  can  C2  SoS  support  warfighters  engaged  in 
complex  endeavors. 

Based  on  the  insights  into  the  problem  domain  and  potential  solutions,  there  are  areas 
that  need  further  study.  The  team  believed  the  C4I  acquisition  and  testing  communities  would 
benefit  from  a  dedicated  Joint  C4I  SoS  manager  to  provide  consistency  in  an  evolving 
environment.  Their  role  could  include  documenting  C4I  SoS  capabilities,  long-range  SoS 
capabilities  planning,  and  testing  requirements  management;  supporting  developmental  and 
operational  testing;  and  addressing  ad  hoc  SoS  configuration,  resulting  from  new  threats  and 
concepts  (Acosta  et  al.,  2007).  These  roles  represent  overlap  between  the  acquisition 
community  and  those  responsible  for  communicating  needed  capabilities  to  them.  It  is  hoped 
that  codifying  the  relationship  between  the  Joint  C2  Capability  Portfolio  Manager  and  the  C2 
FCB  will  be  a  move  in  this  direction. 
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Next,  as  changes  to  the  SoS  configuration  are  made,  the  likelihood  of  capability  failures 
increases.  The  JC3M  team  believes  risk  management  strategies  should  be  developed  and 
applied  to  the  C4I  SoS.  The  JC3M  team’s  preliminary  list  of  risks  includes  the  lack  of  a  single 
entity  responsible  for  SoS  performance;  the  lack  of  an  objective,  repeatable,  and  methodical 
approach  to  address  individual  system  problems  impacting  SoS  functionality;  varied  levels  of 
maturity  of  systems  within  the  C4I  SoS  architecture;  and  varied  interfaces  between  individual 
systems. 

Finally,  systems  that  are  components  of  the  C4I  SoS  have  their  capabilities  defined  as  if 
they  exist  in  a  vacuum,  and  their  impact  on  C4I  SoS  capabilities  is  generally  not  considered. 

The  DoD  C4I  SoS  acquisition  process  should  require  component  system  sponsors  to  define  C4I 
SoS  level  effects;  establish  a  funding  line  for  SoS  testing;  and  include  SoS  effectiveness  testing 
as  part  of  operational  testing  (Acosta  et  al.,  2007). 
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